{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing AI Platform Training Service\n",
    "**Learning Objectives:**\n",
    "  - Learn how to make code compatible with AI Platform Training Service\n",
    "  - Train your model using cloud infrastructure via AI Platform Training Service\n",
    "  - Deploy your model behind a production grade REST API using AI Platform Training Service\n",
    "\n",
    "## Introduction\n",
    "\n",
    "In this notebook we'll make the jump from training and predicting locally, to do doing both in the cloud. We'll take advantage of Google Cloud's [AI Platform Training Service](https://cloud.google.com/ai-platform/). \n",
    "\n",
    "AI Platform Training Service is a managed service that allows the training and deployment of ML models without having to provision or maintain servers. The infrastructure is handled seamlessly by the managed service for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment and run if you need to update your Google SDK\n",
    "# !sudo apt-get update && sudo apt-get --only-upgrade install google-cloud-sdk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make code compatible with AI Platform Training Service\n",
    "In order to make our code compatible with AI Platform Training Service we need to make the following changes:\n",
    "\n",
    "1. Upload data to Google Cloud Storage \n",
    "2. Move code into a Python package\n",
    "3. Modify code to read data from and write checkpoint files to GCS "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload data to Google Cloud Storage (GCS)\n",
    "\n",
    "Cloud services don't have access to our local files, so we need to upload them to a location the Cloud servers can read from. In this case we'll use GCS.\n",
    "\n",
    "Specify your project name and bucket name in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT = \"cloud-training-demos\"  # Replace with your PROJECT\n",
    "BUCKET = \"cloud-training-bucket\"  # Replace with your BUCKET\n",
    "REGION = \"us-central1\"            # Choose an available region for AI Platform Training Service\n",
    "TFVERSION = \"1.14\"                # TF version for AI Platform Training Service to use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jupyter allows the subsitution of python variables into bash commands when using the `!<cmd>` format.\n",
    "It is also possible using the `%%bash` magic but requires an [additional parameter](https://stackoverflow.com/questions/19579546/can-i-access-python-variables-within-a-bash-or-script-ipython-notebook-c). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud config set project {PROJECT}\n",
    "!gcloud storage buckets create --location={REGION} gs://{BUCKET}\n",    "!gcloud storage cp *.csv gs://{BUCKET}/taxifare/smallinput/"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Move code into a python package\n",
    "\n",
    "When you execute a AI Platform Training Service training job, the service zips up your code and ships it to the Cloud so it can be run on Cloud infrastructure. In order to do this AI Platform Training Service requires your code to be a Python package.\n",
    "\n",
    "A Python package is simply a collection of one or more `.py` files along with an `__init__.py` file to identify the containing directory as a package. The `__init__.py` sometimes contains initialization code but for our purposes an empty file suffices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Package Directory and \\_\\_init\\_\\_.py\n",
    "\n",
    "The bash command `touch` creates an empty file in the specified location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "mkdir taxifaremodel\n",
    "touch taxifaremodel/__init__.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Paste existing code into model.py\n",
    "\n",
    "A Python package requires our code to be in a .py file, as opposed to notebook cells. So we simply copy and paste our existing code for the previous notebook into a single file.\n",
    "\n",
    "The %%writefile magic writes the contents of its cell to disk with the specified name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 1**\n",
    "\n",
    "In the cell below, write the content of the `model.py` to the file `taxifaremodel/model.py`. This will allow us to package the model we \n",
    "developed in the previous labs so that we can deploy it to AI Platform Training Service. You'll also need to reuse the input functions and the `EvalSpec`, `TrainSpec`, `RunConfig`, etc. that we implemented in the previous labs.\n",
    "\n",
    "Complete all the TODOs in the cell below by copy/pasting the code we developed in the previous labs. This will write all the necessary components we developed in our notebook to a single `model.py` file. \n",
    "\n",
    "Once we have the code running well locally, we will execute the next cells to train and deploy your packaged model to AI Platform Training Service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile taxifaremodel/model.py\n",
    "# TODO: Your code goes here. Import the necessary libraries (e.g. tensorflow, etc)\n",
    "\n",
    "CSV_COLUMN_NAMES = # TODO: Your code goes here\n",
    "CSV_DEFAULTS = # TODO: Your code goes here\n",
    "FEATURE_NAMES = # TODO: Your code goes here\n",
    "\n",
    "def parse_row(row):\n",
    "    # TODO: Your code goes here\n",
    "    return features, label\n",
    "\n",
    "def read_dataset(csv_path):\n",
    "    # TODO: Your code goes here\n",
    "    return dataset\n",
    "\n",
    "def train_input_fn(csv_path, batch_size = 128):\n",
    "    # TODO: Your code goes here\n",
    "    return dataset\n",
    "\n",
    "def eval_input_fn(csv_path, batch_size = 128):\n",
    "    # TODO: Your code goes here\n",
    "    return dataset\n",
    "  \n",
    "def serving_input_receiver_fn():\n",
    "    # TODO: Your code goes here\n",
    "    return tf.estimator.export.ServingInputReceiver(features = features, receiver_tensors = receiver_tensors)  \n",
    "    \n",
    "def my_rmse(labels, predictions):\n",
    "    # TODO: Your code goes here\n",
    "    return {\"rmse\": tf.metrics.root_mean_squared_error(labels = labels, predictions = pred_values)}\n",
    "\n",
    "def create_model(model_dir, train_steps):\n",
    "    # TODO: Your code goes here\n",
    "    return model\n",
    "\n",
    "def train_and_evaluate(params):\n",
    "    OUTDIR = params[\"output_dir\"]\n",
    "    TRAIN_DATA_PATH = params[\"train_data_path\"]\n",
    "    EVAL_DATA_PATH = params[\"eval_data_path\"]\n",
    "    TRAIN_STEPS = params[\"train_steps\"]\n",
    "\n",
    "    model = # TODO: Your code goes here. \n",
    "\n",
    "    train_spec = # TODO: Your code goes here\n",
    "    \n",
    "    exporter = # TODO: Your code goes here\n",
    "\n",
    "    eval_spec = # TODO: Your code goes here\n",
    "\n",
    "    tf.logging.set_verbosity(tf.logging.INFO) \n",
    "    shutil.rmtree(path = OUTDIR, ignore_errors = True)\n",
    "\n",
    "    tf.estimator.train_and_evaluate(estimator = model, train_spec = train_spec, eval_spec = eval_spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modify code to read data from and write checkpoint files to GCS \n",
    "\n",
    "If you look closely above, you'll notice two changes to the code\n",
    "\n",
    "1. The input function now supports reading a list of files matching a file name pattern instead of just a single CSV\n",
    "  - This is useful because large datasets tend to exist in shards.\n",
    "2. The train and evaluate portion is wrapped in a function that takes a parameter dictionary as an argument.\n",
    "  - This is useful because the output directory, data paths and number of train steps will be different depending on whether we're training locally or in the cloud. Parametrizing allows us to use the same code for both.\n",
    "\n",
    "We specify these parameters at run time via the command line. Which means we need to add code to parse command line parameters and invoke `train_and_evaluate()` with those params. This is the job of the `task.py` file. \n",
    "\n",
    "Exposing parameters to the command line also allows us to use AI Platform Training Service's automatic hyperparameter tuning feature which we'll cover in a future lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 2**\n",
    "\n",
    "Add two additional command line parameter parsers to the list we've started below. You should add code to parse command line parameters for the `output_dir` and the `job-dir`. Look at the examples below to make sure you have the correct format, including a `help` description and `required` specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile taxifaremodel/task.py\n",
    "import argparse\n",
    "import json\n",
    "import os\n",
    "\n",
    "from . import model\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    \n",
    "    parser = argparse.ArgumentParser()\n",
    "    \n",
    "    parser.add_argument(\n",
    "        \"--train_data_path\",\n",
    "        help = \"GCS or local path to training data\",\n",
    "        required = True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--train_steps\",\n",
    "        help = \"Steps to run the training job for (default: 1000)\",\n",
    "        type = int,\n",
    "        default = 1000\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        \"--eval_data_path\",\n",
    "        help = \"GCS or local path to evaluation data\",\n",
    "        required = True\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        # TODO: Your code goes here\n",
    "    )\n",
    "    parser.add_argument(\n",
    "        # TODO: Your code goes here\n",
    "    )\n",
    "    args = parser.parse_args().__dict__\n",
    "\n",
    "    model.train_and_evaluate(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train using AI Platform Training Service (local)\n",
    "\n",
    "AI Platform Training Service comes with a local test tool ([`gcloud ai-platform local train`](https://cloud.google.com/sdk/gcloud/reference/ml-engine/local/train)) to ensure we've packaged our code directly. It's best to first run that for a few steps before trying a Cloud job. \n",
    "\n",
    "The arguments before `-- \\` are for AI Platform Training Service\n",
    "- package-path: speficies the location of the Python package\n",
    "- module-name: specifies which `.py` file should be run within the package. `task.py` is our entry point so we specify that\n",
    "\n",
    "The arguments after `-- \\` are sent to our `task.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "!gcloud ai-platform local train \\\n",
    "    --package-path=taxifaremodel \\\n",
    "    --module-name=taxifaremodel.task \\\n",
    "    -- \\\n",
    "    --train_data_path=taxi-train.csv \\\n",
    "    --eval_data_path=taxi-valid.csv  \\\n",
    "    --train_steps=1 \\\n",
    "    --output_dir=taxi_trained "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train using AI Platform Training Service (Cloud)\n",
    "\n",
    "To submit to the Cloud we use [`gcloud ai-platform jobs submit training [jobname]`](https://cloud.google.com/sdk/gcloud/reference/ml-engine/jobs/submit/training) and simply specify some additional parameters for AI Platform Training Service:\n",
    "- jobname: A unique identifier for the Cloud job. We usually append system time to ensure uniqueness\n",
    "- job-dir: A GCS location to upload the Python package to\n",
    "- runtime-version: Version of TF to use. Defaults to 1.0 if not specified\n",
    "- python-version: Version of Python to use. Defaults to 2.7 if not specified\n",
    "- region: Cloud region to train in. See [here](https://cloud.google.com/ml-engine/docs/tensorflow/regions) for supported AI Platform Training Service regions\n",
    "\n",
    "Below the `-- \\` note how we've changed our `task.py` args to be GCS locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "OUTDIR = \"gs://{}/taxifare/trained_small\".format(BUCKET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
"!gcloud storage rm --recursive --continue-on-error {OUTDIR} # start fresh each time\n",    "!gcloud ai-platform jobs submit training taxifare_$(date -u +%y%m%d_%H%M%S) \\\n",
    "    --package-path=taxifaremodel \\\n",
    "    --module-name=taxifaremodel.task \\\n",
    "    --job-dir=gs://{BUCKET}/taxifare \\\n",
    "    --python-version=3.5 \\\n",
    "    --runtime-version={TFVERSION} \\\n",
    "    --region={REGION} \\\n",
    "    -- \\\n",
    "    --train_data_path=gs://{BUCKET}/taxifare/smallinput/taxi-train.csv \\\n",
    "    --eval_data_path=gs://{BUCKET}/taxifare/smallinput/taxi-valid.csv  \\\n",
    "    --train_steps=1000 \\\n",
    "    --output_dir={OUTDIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can track your job and view logs using [cloud console](https://console.cloud.google.com/mlengine/jobs). It will take 5-10 minutes to complete. **Wait until the job finishes before moving on.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy model\n",
    "\n",
    "Now let's take our exported SavedModel and deploy it behind a REST API. To do so we'll use AI Platform Training Service's managed TF Serving feature which auto-scales based on load."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud storage ls gs://{BUCKET}/taxifare/trained_small/export/exporter"   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AI Platform Training Service uses a model versioning system. First you create a model folder, and within the folder you create versions of the model. \n",
    "\n",
    "Note: You will see an error below if the model folder already exists, it is safe to ignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VERSION='v1'\n",
    "!gcloud ai-platform models create taxifare --region us-central1\n",
    "!gcloud ai-platform versions delete {VERSION} --model taxifare --quiet\n",
    "!gcloud ai-platform versions create {VERSION} --model taxifare \\\n",
"    --origin $(gcloud storage ls gs://{BUCKET}/taxifare/trained_small/export/exporter | tail -1) \\\n",    "    --python-version=3.5 \\\n",
    "    --runtime-version {TFVERSION}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Online prediction\n",
    "\n",
    "Now that we have deployed our model behind a production grade REST API, we can invoke it remotely. \n",
    "\n",
    "We could invoke it directly calling the REST API with an HTTP POST request [reference docs](https://cloud.google.com/ml-engine/reference/rest/v1/projects/predict), however AI Platform Training Service provides an easy way to invoke it via command line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke prediction REST API via command line\n",
    "First we write our prediction requests to file in json format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile ./test.json\n",
    "{\"dayofweek\": 1, \"hourofday\": 0, \"pickuplon\": -73.885262, \"pickuplat\": 40.773008, \"dropofflon\": -73.987232, \"dropofflat\": 40.732403}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use [`gcloud ai-platform predict`](https://cloud.google.com/sdk/gcloud/reference/ml-engine/predict) and specify the model name and location of the json file. Since we don't explicitly specify `--version`, the default model version will be used. \n",
    "\n",
    "Since we only have one version it is already the default, but if we had multiple model versions we can designate the default using [`gcloud ai-platform versions set-default`](https://cloud.google.com/sdk/gcloud/reference/ml-engine/versions/set-default) or using [cloud console](https://pantheon.corp.google.com/mlengine/models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud ai-platform predict --model=taxifare --json-instances=./test.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Invoke prediction REST API via python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 3**\n",
    "\n",
    "In the cell below, use the Google Python client library to query the model you just deployed on AI Platform Training Service. Find the estimated taxi fare for a ride with the following properties\n",
    "- ride occurs on Monday\n",
    "- at 8:00 am\n",
    "- pick up at (40.773, -73.885)\n",
    "- drop off at (40.732, -73.987)\n",
    "\n",
    "Have a look at this post and examples on [\"Using the Python Client Library\"](https://cloud.google.com/ml-engine/docs/tensorflow/python-client-library) and [\"Getting Online Predictions\"](https://cloud.google.com/ml-engine/docs/tensorflow/online-predict) from Google Cloud."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from googleapiclient import discovery\n",
    "from oauth2client.client import GoogleCredentials\n",
    "import json\n",
    "\n",
    "credentials = # TODO: Your code goes here\n",
    "api = # TODO: Your code goes here\n",
    "\n",
    "request_data = {\"instances\":\n",
    "  [\n",
    "      {\n",
    "        # TODO: Your code goes here\n",
    "      }\n",
    "  ]\n",
    "}\n",
    "\n",
    "parent = # TODO: Your code goes here\n",
    "\n",
    "response = # TODO: Your code goes here\n",
    "\n",
    "print(\"response = {0}\".format(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Challenge exercise\n",
    "\n",
    "Modify your solution to the challenge exercise in e_traineval.ipynb appropriately. Make sure that you implement training and deployment. Increase the size of your dataset by 10x since you are running on the cloud. Does your accuracy improve?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2022 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
